Skip to content

feat(semsim): explicit category/taxon filters + search-by-profile - #1375

Open
kevinschaper wants to merge 3 commits into
mainfrom
feature/explicit-semsim-filters
Open

feat(semsim): explicit category/taxon filters + search-by-profile#1375
kevinschaper wants to merge 3 commits into
mainfrom
feature/explicit-semsim-filters

Conversation

@kevinschaper

Copy link
Copy Markdown
Member

Makes semantic-similarity search explicitly filterable and profile-driven, so it can
target arbitrary entity sets — in particular MMRRC mouse genotypes scored against
human disease. Depends on monarch-initiative/koza#245 (closure_size covering
genotypes) for those entities to have profile sizes in monarch-kg.

Commits

  • feat(semsim): explicit category/taxon filters, replacing the prefix proxy
    ducksim now takes explicit category / taxon / entity filters instead of
    inferring scope from a CURIE-prefix heuristic. Adds SemsimSearchFilter /
    request models and a phenotype-level termset-search rewrite. (ducksim.py,
    semsim.py, additional_models.py, ducksim_service.py, semsim_service.py, + tests)
  • docs(ducksim): closure_size is baked upstream now; keep the fallback as a shim
    the on-the-fly closure_size compute becomes a documented fallback now that koza
    bakes it into monarch-kg.
  • feat(semsim): search-by-profile, driven by an entity's own phenotypes
    search_by_profile scores an entity (gene / disease / genotype / MMRRC strain)
    against the corpus using its own phenotype set. (+ unit + router tests)

Why

The prefix-proxy filter couldn't express "score these mouse genotypes against human
disease," which is exactly the MMRRC archiving use case. Explicit category/taxon/entity
filters plus profile-driven search make that a first-class query.

🤖 Generated with Claude Code

https://claude.ai/code/session_01PTYEKvvmBKgaK3893Kxog2

Kevin Schaper and others added 3 commits July 24, 2026 01:46
…roxy

The similarity search API selected entities by a CURIE prefix standing in for
"things of some category in some species". That only works while prefix,
category and taxon coincide one-to-one, and they do not: mouse models are split
across the MGI and MMRRC prefixes so no group can name them, while MGI alone
covers mouse genes, genotypes and variants at once.

Search now takes explicit `categories` / `taxa` / `prefixes`, which AND
together. GET /semsim/filters enumerates the valid combinations from the KG
rather than a hand-maintained enum, and POST /search takes a `filter` object.
SemsimSearchGroup keeps working: it expands to the explicit filters it always
implicitly meant.

Enabling changes:

- DEFAULT_ASSOCIATIONS selects by predicate rather than by association
  category. The old gene+disease allowlist structurally hid all 589,030
  GenotypeToPhenotypicFeature edges -- every MGI and MMRRC mouse model -- plus
  variants and cases, with no way for a caller to opt in. Because the pool is
  now heterogeneous, the legacy groups must state their category to keep their
  old meaning; verified they still return a single category and the same top
  hits (MECP2/Mecp2/Rett syndrome for a Rett query).

- _termset_search scores at the phenotype level instead of per (entity,
  phenotype). A phenotype's similarity to a query term does not depend on which
  entity carries it, so the old query re-expanded each phenotype's ancestors
  once per annotated entity: across mouse genotypes, 447,110 association rows
  covering 11,530 distinct MP terms, ~39x redundant. Scoring all 81,151 mouse
  genotypes against Rett syndrome goes from 288.7s to 2.3s (~125x). Verified
  equivalent, not assumed: both paths return all 81,151 entities with a maximum
  absolute score difference of 2.0e-15. Exact because `closure` holds no
  duplicate (subject, object) pairs, so count(*) and count(DISTINCT a) agree.

- _flat computes closure sizes for entities the baked `closure_size` table
  omits. It covers genes and diseases but no genotypes, and _flat inner-joined
  it, so hybrid_search -- the production default -- returned an empty result
  set for every mouse model with no error at all.

Filtering on a `nodes` table lacking category/in_taxon now raises instead of
matching nothing, so a filter that cannot be honored never looks like a
legitimately empty result. Same reasoning for SemsimianService, which can only
express a single prefix and now refuses anything else rather than silently
answering with an unfiltered superset.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PTYEKvvmBKgaK3893Kxog2
…as a shim

koza now bakes a closure size for every entity with a has_phenotype edge, so
_flat's computed fallback is no longer what makes genotype search work. Keep it
for artifacts built before that fix, where degrading to a slower correct answer
still beats silently returning nothing.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PTYEKvvmBKgaK3893Kxog2
Case entities were already reachable as search TARGETS once the association pool
was selected by predicate rather than category -- filter to biolink:Case and a
termset search ranks patients. The missing half is using a Case as a search
SOURCE: a phenopacket has ~20 HPO terms, and no one is going to retype them to
ask "which mouse models this patient".

search_by_profile takes any phenotype-annotated entity id, reads its profile
from the KG, and searches with an explicit target filter. Because the profile
comes from the same _assoc pool for every entity type, one call expresses:

  Case  -> Disease   likely diagnosis for a patient
  Case  -> Genotype  mouse models for a patient        (needs mouse taxon filter)
  Genotype -> Case   patients resembling a mouse model (the MMRRC<->phenopacket link)

Engine gets a thin `profile(entity)` accessor; the service composes it with the
existing search and returns (query_profile, results) so a caller can weight each
result by how many terms drove it. The endpoint is guarded to the ducksim
backend -- semsimian has no KG to read a profile from -- returning 501 rather
than a confusing failure, and 404 when the entity has no phenotypes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PTYEKvvmBKgaK3893Kxog2
@netlify

netlify Bot commented Jul 28, 2026

Copy link
Copy Markdown

Deploy Preview for monarch-app ready!

Name Link
🔨 Latest commit 54a49fc
🔍 Latest deploy log https://app.netlify.com/projects/monarch-app/deploys/6a683f8cdae5540009ba02f0
😎 Deploy Preview https://deploy-preview-1375--monarch-app.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@codecov

codecov Bot commented Jul 28, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 82.26950% with 25 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.30%. Comparing base (d0ea0a3) to head (54a49fc).
⚠️ Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
backend/src/monarch_py/api/semsim.py 42.10% 11 Missing ⚠️
backend/src/monarch_py/service/semsim_service.py 0.00% 6 Missing ⚠️
backend/src/monarch_py/api/additional_models.py 83.33% 5 Missing ⚠️
backend/src/monarch_py/service/ducksim.py 93.75% 3 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1375      +/-   ##
==========================================
- Coverage   83.34%   83.30%   -0.05%     
==========================================
  Files         126      126              
  Lines        6648     6779     +131     
==========================================
+ Hits         5541     5647     +106     
- Misses       1107     1132      +25     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@kevinschaper

Copy link
Copy Markdown
Member Author

Part of #1376 (umbrella: ducksim-only similarity features). This PR delivers the explicit corpus filters, search_by_profile, and directionality items on that list; the remaining roadmap items are tracked there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant